Modalities for authorizing access when operating an automated assistant enabled vehicle

ABSTRACT

Implementations relate to enabling of authorization of certain automated assistant functions via one or more modalities available within a vehicle. Implementations can eliminate wasting of computational and communication resources by at least allowing other users to authorize execution of certain input commands from a user, without requesting the user to re-submit the commands. The vehicle can include a computing device that provides access to restricted data, which can be accessed in order for an action to be performed by the automated assistant. However, when a restricted user requests that the automated assistant perform an action involving accessing the restricted data, the automated assistant can be authorized or unauthorized to proceed with fulfilling the request via a modality controlled by an unrestricted user. The unrestricted user can also cause contextual restrictions to be established for limiting functionality of the automated assistant during a trip, for certain types of requests, and/or for certain passengers.

BACKGROUND

Humans may engage in human-to-computer dialogs with interactive softwareapplications referred to herein as “automated assistants” (also referredto as “digital agents,” “chatbots,” “interactive personal assistants,”“intelligent personal assistants,” “assistant applications,”“conversational agents,” etc.). For example, humans (which when theyinteract with automated assistants may be referred to as “users”) mayprovide commands and/or requests to an automated assistant using spokennatural language input (i.e., utterances), which may in some cases beconverted into text and then processed, and/or by providing textual(e.g., typed) natural language input.

Automated assistants can be integrated into a variety of electronicdevices, including vehicles. Unlike other computing devices such asmobile phones, vehicles are generally in motion over a large area, andthus are more susceptible to bandwidth restrictions duringcommunications with an outside server. This can in part result from thevehicle moving through areas that do not provide adequate networkcoverage. This can negatively affect automated assistant operations,which may involve multiple round trips between a vehicle computingdevice and a remote server.

Automated assistants may have access to publicly-available data as wellas user-specific data, which can be associated with a personal useraccount served by the automated assistant. An automated assistantserving multiple users may have multiple accounts with different dataavailable for each account. Furthermore, while the automated assistantis serving one account, the automated assistant may not have access tothe user-specific data of another account. Thus, if one user makes arequest to an automated assistant, and responding to the requestinvolves accessing a second user account, the automated assistant maynot be able to complete the request without prompting the second user tolog into their account and repeat the request. As a result,computational and communication resources, such as network bandwidth andchannel usage time, can be consumed by increasing a number ofinteractions between the vehicle computing device and the server.

SUMMARY

Implementations described herein relate to limiting vehicle automatedassistant responsiveness according to a restriction(s) that is used todetermine whether certain input commands and/or certain users arerestricted in certain vehicle contexts. Furthermore, implementationsdescribed herein allow for other users to override certain restrictionsby providing authorization via an input to the vehicle computing deviceor other computing device. Allowing other users to override suchrestrictions can preserve computational resources, as less processingresources and network bandwidth would be consumed when a restricted userdoes not have to rephrase and resubmit certain inputs in a way thatwould make the inputs permissible. As an example, a passenger thatprovides a spoken input to a vehicle automated assistant such as“Assistant, send a message to Karen,” may be denied because thepassenger is not the owner of the vehicle or otherwise permitted toaccess contacts accessible to the vehicle automated assistant. As aresult, the vehicle automated assistant can provide a response such as“I'm sorry, you are not authorized for such commands,” and the passengerwould have to rephrase and resubmit the spoken input as, for example,“Ok, Assistant, send a message to 971-555-3141.” Such a dialog sessionbetween the passenger and the vehicle automated assistant can wastecomputational resources as the subsequent spoken input would have to beconverted to audio data, transmitted over a network, and subsequentlyprocessed. In a situation where available bandwidth is limited orvariable, such as for example in a moving vehicle, this might beparticularly undesirable since the channel over which data iscommunicated from the assistant device, over the network, may need to beused for longer than desirable. The length of time such a channel isused might impact not only operations of the automated assistant butalso other software applications which rely on the network to transmitand receive information. Such software applications may, for example, bepresent in the same device as the automated assistant (e.g. otherin-vehicle software applications). However, implementations providedherein can eliminate such wasting of computational and communicationresources by at least allowing other users to authorize execution ofcertain input commands from a user, without requesting the user tore-submit the commands.

In some implementations, a vehicle computing device and/or an automatedassistant can operate according to one or more different restrictionsfor restricting access to commands and/or data that would otherwise beaccessible via the vehicle computing device and/or the automatedassistant. A restriction can characterize one or more particularcommands, data, type(s) of data, and/or any other input(s) and/oroutput(s) that can be associated with an automated assistant, therebydefining certain information that is available to other users via theautomated assistant and/or the vehicle computing device. When a userprovides a spoken utterance corresponding to a particular commandcharacterized by a restriction, the automated assistant can respondaccording to any restriction that is associated with the user and/or theparticular command. As an example, when a user provides a spokenutterance that corresponds to data that originated at a computing deviceowned by another user, the spoken utterance can satisfy a criterion forrestricting access to such data. However, in response to receiving thespoken utterance, the automated assistant can determine that thecriterion is satisfied and await authorization from the other user. Theauthorization can be provided by the other user to the vehicle computingdevice and/or a separate computing device via another spoken utteranceand/or any other input capable of being received at a computing device.In some implementations, a vehicle that includes the vehicle computingdevice can include an interface, such as a button (e.g., on a steeringwheel of the vehicle), that the other user can interact with (e.g.,depress the button) in order to indicate authorization to the automatedassistant. In response to the automated assistant receivingauthorization from the other user, the automated assistant can proceedwith executing the command provided by the user, without necessarilyrequesting further input from the user.

In some implementations, another user can limit a passenger fromaccessing certain data while the other user and the passenger are ridingin the vehicle. Specifically, the other user can limit access to certaindata while the vehicle is navigating along a particular route and/or toa particular destination. Therefore, when the vehicle completes theroute, and/or arrives at the particular destination, a restriction onaccess to the particular data and/or for the passenger can be released,thereby allowing the passenger to subsequently access such data. Forinstance, when the other user is driving the vehicle and the passengeris riding in the vehicle, the passenger can provide a spoken utteranceto an automated assistant interface of the vehicle. The spoken utterancecan be, “Assistant, call Aunt Lucy.” In response, and because the spokenutterance includes a request that will result in accessing contactinformation of the user, the automated assistant can await authorizationfrom the user before fulfilling the request. However, in order toeliminate having to repeatedly authorize or not authorize requestsoriginating from the passenger, the user can provide another spokenutterance such as, “Assistant, do not respond to the passenger for theremainder of this trip.” In response, the automated assistant can causerestriction data to be generated for limiting access to one or moreservices (e.g., making phone calls) that would otherwise be availablevia the automated assistant. In this way, the user would not have torepeatedly authorize or not authorize the automated assistant to respondto requests from the passenger, thereby eliminating waste ofcomputational resources and network resources. Furthermore, because theaccess restrictions can be set to “reset” at the end of a trip, or uponreaching a destination, the user would not have to explicitly request areset of restrictions, thereby further eliminating waste ofcomputational resources and network resources.

Additionally, or alternatively, the user can limit access to certaindata to a passenger indefinitely and/or for an operational lifetime ofthe vehicle. For instance, subsequent to the passenger providing thespoken utterance, “Assistant, call Aunt Lucy,” and while the automatedassistant is awaiting authorization from the user, the user can providea separate spoken utterance such as, “Assistant, never respond to theuser.” In response, the automated assistant can cause restriction datato be generated for indefinitely (or for an operational lifetime of thevehicle, the vehicle computing device, and/or the automated assistant)limiting access to one or more services that would otherwise beavailable to a particular user via the automated assistant.

In some implementations, depending on an occupancy of the vehicle, theautomated assistant and/or the vehicle computing device can operateaccording to an operating mode that limits access to the automatedassistant and/or the vehicle computing device for certain passengers. Asan example, when a user is the only person occupying a vehicle, avehicle computing device and/or an automated assistant that isaccessible via the vehicle computing device, can operate according to afirst operating mode. The occupancy can be determined based on an outputof one or more sensors of the vehicle, the vehicle computing device,and/or any other device that can provide an output from which occupancycan be estimated. The first operating mode can be selected based on theoccupancy and can provide the user access to a first set of services,data, and/or commands, associated with the automated assistant. When theoccupancy is determined to include more than the user, such as when theuser is driving with one or more passengers (e.g., a parent driving withmultiple children as passengers), a second operating mode can beselected. In accordance with the second operating mode, the user canstill access the first set of services, data, and/or commands—however,the one or more passengers would only be able to access a second set ofservices, data, and/or commands. In some implementations, the second setcan be different than the first set, and/or the second set can be areduced subset relative to the first set. For example, push “talk”button on head unit, when only a driver (e.g., an unrestricted user) isin the vehicle, can respond with private data without any furtherauthorization. However, if the “talk” button on the head unit is pushedwhen a passenger (e.g., a restricted user) is in the vehicle with thedriver, the automated assistant request further authorization to respondto someone (e.g., the passenger) pressing the “talk” button on the headunit.

While the second operating mode (e.g., a shared operating mode) isactive, a passenger can attempt to access a service, data, and/or acommand that is exclusively provided in the first set, and not thesecond set. In order to permit such access, the user (e.g., the driver)can provide one or more inputs to the automated assistant and/or thevehicle computing device, in order to authorize such access. The usercan provide, for example, an input to an interface such as a buttonand/or touch display panel, which can be located approximately withinreach of a driver of the vehicle (e.g., a button on a steering wheel, atouch display panel integral to a dashboard and/or console). In someimplementations, the authorizing input can be provided in response tothe automated assistant soliciting authorization from the user (e.g.,“Sorry, I need authorization to do that . . . [authorizing inputsubsequently received]”). Alternatively, the automated assistant canbypass soliciting the user for authorization, and, rather, passivelyawait to respond to a request from a passenger until the user providesan authorizing input. However, if the user elects to have theirautomated assistant and/or their vehicle computing device operateaccording to a third operating mode. In the third operating mode, inwhich no option to provide such authorization is available, theautomated assistant and/or the vehicle computing device can operate suchthat availability of certain operations, data, and/or services arelimited for some passengers (at least relative to a user that is aprimary and/or “master” user with respect to the automated assistantand/or the vehicle computing device).

In some implementations, an automated assistant can perform one or moreautomated assistant routines. An automated assistant routine cancorrespond to set and/or sequence of actions performed and/orinitialized by the automated assistant in response to a user providing aparticular input. As an example, the user can provide a spoken utterancesuch as, “Assistant, let's go to work,” when the user enters theirvehicle, in order to cause the automated assistant to perform a “Goingto Work” routine. The “Going to Work” routine can involve the automatedassistant causing the vehicle computing device to render graphical datacorresponding to a daily schedule of the user, render audio datacorresponding to a podcast selected by the user, and generate a messageto a spouse of the user indicating that the user is headed to work(e.g., “Hi Billy, I'm headed to work.”). In some instances, however, apassenger of the vehicle can provide the spoken utterance, “Assistant,let's go to work.” Depending on the mode that the vehicle computingdevice and/or the automated assistant is operating in, the automatedassistant can request that the driver, or other authorized user, providepermission to perform one or more actions of a requested routing.

For example, in response to the passenger invoking the “Going to Work”routine, the automated assistant can initialize performance renderingaudio data corresponding to a particular podcast, and also prompt thedriver for authorization to initialize other actions of the routine.Specifically, the vehicle computing device and/or server device canidentify actions of the routine that involve accessing restricted data.In this instance, the vehicle computing device and/or the server devicecan determine that the schedule of the user and the contacts of the user(for sending the message) are restricted data. As a result, duringperformance of the routine, the driver can be prompted one or more timesto give permission to execute any action(s) involving accessingrestricted data. If the driver gives authorization (e.g., via anassistant invocation task), by speaking an invocation phrase (e.g., “Ok,Assistant.”) or interacting with an interface (e.g., pressing a button),the routine can be completed. For instance, and in furtherance of theaforementioned example, the message can be sent to the spouse and/or theschedule of the driver can be rendered audibly.

However, if authorization is not provided by the driver (e.g., thedriver does not perform an assistant invocation task), the automatedassistant can bypass performance of such actions. In someimplementations, when the driver does not provide authorization tocomplete the actions, alternative actions can be provided as options tothe passenger. For instance, instead of audibly rendering the scheduleof the driver, the automated assistant can audibly render publicinformation about events that are occurring in the nearby geographicregion. Additionally, or alternatively, instead of sending a message toa spouse of the driver, the automated assistant can audibly prompt thepassenger regarding whether they would like to have a messagetransmitted via their own account (e.g., “Would you like to login, inorder to send a message?”). In this way, restrictions on data of thedriver would be enforced while simultaneously providing assistance to apassenger who may be in the vehicle due to, for example, participationin a ride-sharing activity.

The above description is provided as an overview of some implementationsof the present disclosure. Further description of those implementations,and other implementations, are described in more detail below.

Other implementations may include a non-transitory computer readablestorage medium storing instructions executable by one or more processors(e.g., central processing unit(s) (CPU(s)), graphics processing unit(s)(GPU(s)), and/or tensor processing unit(s) (TPU(s)) to perform a methodsuch as one or more of the methods described above and/or elsewhereherein. Yet other implementations may include a system of one or morecomputers and/or one or more robots that include one or more processorsoperable to execute stored instructions to perform a method such as oneor more of the methods described above and/or elsewhere herein.

It should be appreciated that all combinations of the foregoing conceptsand additional concepts described in greater detail herein arecontemplated as being part of the subject matter disclosed herein. Forexample, all combinations of claimed subject matter appearing at the endof this disclosure are contemplated as being part of the subject matterdisclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates a view of a user authorizing continued performanceof a routine initialized by another user, subsequent to the other userinitializing a routine that involves accessing restricted data.

FIG. 1B illustrates a view of a user setting a contextual restriction onaccess to data, services, an automated assistant, and/or any otherinformation or module that can be accessed via the computing device.

FIG. 1C illustrates a view of the user establishing restrictions oncertain types of requests that can be provided by passengers and/orother persons that can occupy the vehicle.

FIG. 2 illustrates a system for allowing a user to employ restrictionson particular interactions between passengers of the vehicle and anautomated assistant, and/or authorize such interactions using one ormore different modalities.

FIG. 3A and FIG. 3B illustrate methods for establishing and enforcingaccess restrictions for an automated assistant based on one or moredifferent factors.

FIG. 4 is a block diagram of an example computer system.

DETAILED DESCRIPTION

FIG. 1A illustrates a view 100 of a user 108 authorizing continuedperformance of a routine initialized by another user 110, subsequent tothe other user 110 initializing a routine that involves accessingrestricted data. Restricted data can refer to any data that isassociated with an account of the user 108, and can be accessed byproviding credentials in order to log into the account of the user 108.By allowing the user 108 to authorize such interactions, through one ormore different modalities, network bandwidth and other computationalresources can be preserved, at least by reducing a number ofinteractions between a client device and a server device 126.

In some implementations, the user 108 can be an unrestricted user withrespect to access to restricted data and/or an automated assistant 116,that is accessible via a computing device 114 of a vehicle 112. Thecomputing device 114 can be a vehicle computing device that is connectedto one or more different interfaces, through which the user 108 and/orthe other user 110 can provide inputs to the computing device 114, andreceive output from the computing device 114. For instance, the user 108and/or the other user 110 can perform an assistant invocation task forinvoking the automated assistant 116 and/or an automated assistant 128,that is accessible to the computing device 114 via a network 122, suchas the internet. The assistant invocation task can include speaking aparticular invocation phrase such as, but not limited to, “Ok,Assistant,” and/or a voice characteristic(s) of the user being verifiedin response. For instance, the user can provide a particular, or anyspoken utterance, to the automated assistant interface, and audio datacorresponding to the spoken utterance can be processed to determinewhether one or more voice characteristics of an authorized and/orunrestricted user are embodied in the audio data. When the one or morevoice characteristics of an authorized and/or unrestricted user areembodied in the audio data, authorization can be considered received.

Additionally, or alternatively, the assistant invocation task caninclude the user interacting with an interface of the computing device114 by performing a gesture such as, but not limited to, pressing abutton, tapping a touch interface, providing a fingerprint, looking intoa camera for performing facial recognition, and/or any other gesturethat can invoke an automated assistant. The automated assistant 116 canaccess client data 120 and/or server data 130 when executing certainactions requested by a particular user. Furthermore, depending on anaction that is requested, the automated assistant 116 can access one ormore applications 118 in furtherance of executing an action. In someimplementations, applications and/or data that are accessible to theautomated assistant can be third-party applications and/or third-partydata, at least relative to a party or entity that provides the automatedassistant.

In some implementations, a user can invoke the automated assistantand/or provide authorization to the automated assistant via one or moredifferent interfaces of a vehicle. Furthermore, an effect of interactingwith a particular interface can be contingent upon an occupancy of thevehicle. As an example, when multiple persons are in a vehicle, a buttonon the steering wheel of the vehicle can be designated, by the vehiclecomputing device, as an exclusive touch interface for invoking and/orproviding authorization to the automated assistant. Additionally, oralternatively, when only one person is in the vehicle, the vehiclecomputing device can designate the button on the steering wheel, and/orone or more other buttons (e.g., a button attached to a backseat and/ora touch interface of a display panel), for invoking and/or providingauthorization to the automated assistant.

The automated assistant 116 can perform routines, which can beconfigured by the user 108, the other user 110, and/or can bereconfigured by an entity that provides the automated assistant.Furthermore, a routine can include one or more actions that involveaccessing restricted data, or any data that corresponds to one or moreparticular access rights. For example, when the other user 110 providesa spoken utterance 102 such as, “Assistant, perform the drivers morningcommute routine,” the automated assistant can perform a series ofactions that were previously specified by the user 108. One or more ofthe actions for they requested routine can involve accessing restricteddata, such as calendar data stored in association with an account of theuser 108. For example, the series of actions corresponding to themorning commute routine can include playing a podcast, and/or renderingaudio and/or visual content that is based on calendar data for thecurrent day.

In response to receiving the spoken utterance 102, the computing device114 and/or the server device 126 can determine an occupancy of thevehicle 112. The occupancy can be determined based on sensor data fromone or more sensors that are connected to the vehicle 112 and/or thecomputing device 114. For example, and with permission from the user 108and/or the other user 110, certain sensors, such as one or more of amicrophone, a camera, a motion sensor, a weight sensor, a heat sensor, atactile sensor, and/or any other sensor that can be used to detectoccupancy of an area. When the determined occupancy indicates that morethan one person is located within the vehicle 112, a computing device114, and/or the server device 126, can determine whether the routineincludes one or more actions that involve accessing restricted data. Forexample, the computing device 114 can determine whether the morningcommute routine involves accessing restricted data. Because the morningcommute routine involves accessing calendar data corresponding to theuser 108, the computing device 114 can positively determine that themorning commute routine involves accessing restricted data.

When the computing device 114 and/or the server device 126 haveidentified one or more actions of the routine that involve accessingrestricted data, the computing device 114 and/or the server device 126can cause the automated assistant 116 to initialize performance of oneor more actions of the routine that do not involve accessing restricteddata. For instance, an action of rendering podcast audio can involveaccessing unrestricted data, and therefore can be initialized by theautomated assistant 116 in order to provide an output 104 via thevehicle computing device 114. While they computing device 114 isrendering the output 104 that includes podcast audio, the automatedassistant 116 can provide another output 104 via a speaker of thevehicle 112 and/or a display panel of the vehicle 112, in order toindicate that the other user 110 is not authorized to cause certainactions to be performed. For example, the automated assistant 116 canprovide an output such as, “You are not authorized for some actions.”

The user 108 can acknowledge the output 104 from the automated assistant116, and provide authorization via an interface of the vehicle computingdevice 114. In other words, with or without being prompted by theautomated assistant 116, the user 108 can provide an input in order tocause the automated assistant 116 to proceed with executing restrictedactions of the routine. As an example, the user 108 can provide aninterface authorization 106 by pressing a button on the steering wheelof a vehicle 112, tapping a touch interface of the vehicle 112,providing a spoken utterance to a microphone of the vehicle 112 or aperipheral device, and/or otherwise provide an authorization input tothe computing device 114 and/or the server device 126.

In response to the computing device 114 and/or the server device 126receiving the interface authorization 106 from the user 108, theautomated assistant 116 can continue performing the requested routine.In other words, because the automated assistant 116 had stoppedperformance of the action that involved accessing the calendar data ofthe user 108, the automated assistant 116 can proceed with an action inresponse to the user 108 providing the interface authorization 106.Therefore, in response to the interface authorization 106, the automatedassistant 116 can provide an output 104 such as, “Your schedule for theday is as follows . . . ”

By allowing the user 108 to provide authorization without necessitatingfurther input from the other user 110, network bandwidth can bepreserved over the network 122, and/or any other communication channelinvolved in interactions between the users and the automated assistant116. Furthermore, other computational resources can be preserved, suchas processing resources that would otherwise be expended on processingnatural language input from the other user 110. In some implementations,verification of an identity of the user 108 can be performed prior tothe interface authorization 106. For example, based on one or moresensors of the vehicle 112, the computing device 114 and/or the serverdevice 126 can determine that the user 108 is an unrestricted user withrespect to the automated assistant 116. Such verification can be basedon a voice signature of a previous spoken utterance provided by the user108, facial recognition previously performed when the user 108 and theother user 110 are in the vehicle 112, fingerprint sensing, and/or anyother process through which an identity of the user 108 can be verified,with prior permission from the user 108.

FIG. 1B illustrates a view 140 of a user 108 setting a contextualrestriction on access to data, services, an automated assistant, and/orany other information or module that can be accessed via the computingdevice 114. The user 108 can establish a contextual restriction byproviding an input to an automated assistant 116. When the user 108provides the input, a voice signature of the user 108 can be verified,and one or more accounts associated with the user 108 can be identifiedbased on the voice signature. Furthermore, when the input corresponds toa request to apply a contextual restriction, the automated assistant 116can cause certain settings to be generated and/or modified infurtherance of establishing access restrictions. In someimplementations, the user 108 can request certain access restrictions beenforced when a particular context is temporary, such as when the user108 and the other user 110 are riding in the vehicle 112. Suchrestrictions can be useful in order to eliminate distractions created bypassengers within the vehicle who may be attempting to communicate withthe automated assistant 116. Furthermore, such restrictions can beuseful to eliminate safety risks that may be apparent when inexperiencedusers, such as children, provide requests to the automated assistant 116that can affect an operation of the vehicle 112.

As an example, the other user 110 can provide a spoken utterance 142such as, “Assistant, turn off cruise control.” In response to receivingthe spoken utterance, the automated assistant 116 can determine, basedon audio data characterizing the spoken utterance 142, that the spokenutterance 142 was not provided by the restricted user, and/or wasprovided by an unrestricted user. In response, the automated assistant116 can cause an interface of the computing device 114 to provide anoutput 144 such as, “You are not authorized for that request.” In viewof this denial of the request, and in order to eliminate further wastingof computational resources in processing inputs from the other user 110,the user 108 can provide a separate spoken utterance 146, requesting acontextual restriction be established. Specifically, the user 108 canprovide a spoken utterance 146 such as, “Assistant, do not respond toLuke for the remainder of this trip.”

In response to receiving the spoken utterance 146 from the user 108, theautomated assistant 116 can initialize generation of one or more accessrestrictions, which can be characterized by restriction data. In someimplementations, when the other user 110 is providing a spoken utterancefor the first time to the automated assistant 116, the automatedassistant 116 can generate a guest entry corresponding to the other user110. The guest entry can correspond to a guest account, with which theautomated assistant 116 can also correlate the restriction data. In someimplementations, an identifier for the guest account can be determinedusing one or more inputs from the user 108, such as the spoken utterance146, (which specifies the name of the other user 110 (e.g., Luke)) atyped input which can be provided at a touch display panel of thevehicle 112, and/or any other input that can be used to identify a namefor a user.

In some implementations, restriction data can characterize a context inwhich one or more restrictions apply. For instance, because the user 108specified that the automated assistant 116 should not respond to theother user 110 for the remainder of the trip, a computing device 114and/or the server device 126 can identify navigation data characterizinga current trip of the vehicle 112. In some implementations, client data120 corresponding to one or more applications 118 can be accessed, withpermission from the user 108, in order to determine a current context ofthe user 108, the vehicle 112, and/or the other user 110. For instance,the client data 120 can specify a destination for the vehicle 112, asspecified by the user 108 via a navigation application. Therefore, therestriction data can characterize a restriction on access to theautomated assistant 116 while the vehicle 112 is navigating and/orotherwise driving to the destination. As an example, from the currenttime, after which the user 108 provided the spoken utterance 146, and atime when the vehicle 112 arrives at the destination, the automatedassistant 116 can be unresponsive to one or more requests and/or otherinputs from the other user 110. In this way, computational resources canbe preserved by not requesting authorization during the remainingportion of the trip, but rather, can follow the instructions of the user108 to not be responsive to the other user 110. Furthermore, by allowingthe user 108 to set up contextual restrictions with the automatedassistant, interactions between passengers and the automated assistantcan be streamlined based on preferences of an unrestricted user. Forinstance, rather than constantly requesting authorization from the user108, or awaiting authorization, the automated assistant 116 can dealwith certain inputs from other users according to restrictions specifiedby the user 108. This can result in less frequency of interactions withthe automated assistant 116 during vehicle drive times, therebypreserving stored energy that is available to the vehicle 112, and whichmay not be immediately recharged.

FIG. 1C illustrates a view 160 of the user 108 establishing restrictionson certain types of requests that can be provided by passengers and/orother persons that can occupy the vehicle 112. Specifically, the user108 and/or any other unrestricted user can provide a spoken utterance tothe automated assistant 116 in order to specify certain restrictionsthat the automated assistant 116 will comply with. In someimplementations, the user 108 can provide a request to the automatedassistant 116 in order to prevent the automated assistant 116 fromresponding to certain types of requests from other persons within thevehicle 112. For example, when the user 108 is operating their vehicle112 in furtherance of participating in ride-sharing activity, the user108 can establish certain restrictions that will be enforcedspecifically when the user 108 is operating their vehicle forride-sharing services. In some implementations, the automated assistant116 can determine that the user 108 is operating the vehicle 112 infurtherance of ride-sharing activity by accessing one or moreapplications 118 of the computing device 114, the server device 126,and/or any other computing device that can be associated with the user108.

For example, the user 108 can operate a ride-sharing application duringa particular voyage of the vehicle 112. Client data 120 of the computingdevice 114 can characterize this activity, and the automated assistant116 can access the client data 120 in order to determine that the user108 is engaging in such activity. As a result, the automated assistant116 and/or the automated assistant 128 can operate in accordance withone or more restrictions established for governing interactions with theautomated assistant 116 and/or the automated assistant 128 when the user108 is participating in ride sharing activity.

In some implementations, in order to establish a particular restriction,the user 108 can provide a spoken utterance 166 during an attemptedinteraction between another user 110 and the automated assistant 116.For example, when the user 108 is participating in a ride sharingactivity in which the other user 110 is also participating, the otheruser 110 can provide a spoken utterance 162 such as, “Assistant, what isgoing on today?” In response to receiving the spoken utterance 162, thecomputing device 114 and/or a remote computing device 124, such as aserver device 126, can process audio data corresponding to be spokenutterance 162. Based on the processing, a computing device 114 and/orthe server device 126 can determine that the other user 110 is not anunrestricted user like the user 108. Therefore, in response to receivingthe spoken utterance 162, the automated assistant 116 can provide anoutput 164 such as, “You are not authorized for that information.”

In order to cause a restriction to be generated for restricting accessand/or otherwise limiting responses of the automated assistant 116 insuch situations, the user 108 can provide a request to the automatedassistant 116. Specifically, the user 108 can provide a spoken utterance166 such as, “Assistant, do not respond to anyone else for those typesof requests.” Audio data corresponding to the spoken utterance 166 canbe processed at the computing device 114 and/or the server device 126.Based on the processing, restriction data can be generated forcharacterizing a restriction on one or more particular types of requestthat can be responded to by the automated assistant 116, when one ormore particular types of requests are provided by restricted users,and/or any other user that is not an unrestricted user.

In some implementations, in order to characterize the type of requestembodied by the spoken utterance 162, a context of the spoken utterance162 can be determined from one or more sources of data. For example, thetype of request can be characterized as a request from a passengerduring ride sharing activity. Therefore, when the restriction data isgenerated, the restriction data can characterize a limitation on whatpassengers can request when they are riding in the vehicle 112 and infurtherance of ride sharing activity. Alternatively, or additionally,the type of request can be determined based on data that would have beenaccessed in order to fulfill the request. For example, as provided inFIG. 1C, the other user 110 requested information about what is going ontoday, but if such request had been provided by the user 108, fulfillingthe request would involve accessing calendar your data associated withthe user 108. Therefore, the restriction data can characterize alimitation on requests related to calendar data, as well as a limitationon such requests being provided by a passenger during ride sharingactivity.

In some implementations, restriction data can characterize limitationsthat are based on occupancy and whether the vehicle is being operated infurtherance of ride sharing activities. For instance, when one or moresensors indicate that the vehicle is occupied by multiple persons, and aride sharing application indicates that the vehicle is currently beingused for ride sharing activities, more limitations on the automatedassistant can be enforced, at least relative to when the vehicle isoccupied by multiple persons but not currently being used for ridesharing activities. In this way, for example, stricter limitations onaccess to restricted data can be enforced at times when multiple personsare in the vehicle and the vehicle is operating in furtherance of ridesharing activity, compared to if a user is merely driving with theirfamily or friends (e.g., not ride sharing customers). Furthermore, thisprevents the user from having to provide restriction requests to theautomated assistant each time a new person enters their car while theirride sharing application indicates that they are operating their vehiclein furtherance of ride sharing activities.

In some implementations, and based on the restriction data, theautomated assistant 116 can operate such that calendar data is notavailable to the other user 110, and elect to respond to the other user110 using publicly available data. For example, the automated assistant116 can render an output 164 such as, “There is music at the ConcertVenue, and a basketball game.” In this way, should the other user 110and/or another passenger provide a similar type of request to theautomated assistant 116. The computing device 114 and/or the serverdevice 126 would not have to provide an output 164 indicating a lack ofauthorization, nor would the computing device 114 and/or the serverdevice 126 necessarily have to process input from the user 108. Rather,restriction data that is available can be processed along with anysubsequent spoken utterance in order to determine certain limitationsthat are currently in effect. Based on such limitations, the automatedassistant 116 can render a response to any subsequent user or passenger,without providing an output denying the request, and/or a prompt to theuser 108 requesting authorization.

FIG. 2 illustrates a system 200 for enabling authorization of certainautomated assistant functions via one or more modalities availablewithin a vehicle. The automated assistant 204 can operate as part of anassistant application that is provided at one or more computing devices,such as a computing device 218 and/or a server device 202. A user caninteract with the automated assistant 204 via an assistant interface220, which can be a one or more of a microphone, a camera, a touchscreen display, a user interface, and/or any other apparatus capable ofproviding an interface between a user and an application. For instance,a user can initialize the automated assistant 204 by providing a verbal,textual, or a graphical input to the assistant interface 220 to causethe automated assistant 204 to perform a function (e.g., provide data,control a peripheral device, access an agent, generate an input and/oran output, etc.). The computing device 218 can include a display device,which can be a display panel that includes a touch interface forreceiving touch inputs and/or gestures for allowing a user to controlapplications of the computing device 218 via the touch interface. Insome implementations, computing device 218 can lack a display device,thereby providing an audible user interface output, without providing agraphical user interface output. Furthermore, the computing device 218can provide a user interface, such as a microphone, for receiving spokennatural language inputs from a user. In some implementations, thecomputing device 218 can include a touch interface and can be void of acamera, but can optionally include one or more other sensors.

The computing device 218 and/or other computing devices 234 can be incommunication with the server device 202 over a network 240, such as theinternet. Additionally, the computing device 218 and the other computingdevices 242 can be in communication with each other over a local areanetwork (LAN), such as a Wi-Fi network. The computing device 218 canoffload computational tasks to the server device 202 in order toconserve computational resources at the computing device 218. Forinstance, the server device 202 can host the automated assistant 204,and computing device 218 can transmit inputs received at one or moreassistant interfaces 220 to the server device 202. However, in someimplementations, the automated assistant 204 can be hosted at thecomputing device 218 as a client automated assistant 236.

In various implementations, all or less than all aspects of theautomated assistant 204 can be implemented on the computing device 218.In some of those implementations, aspects of the automated assistant 204are implemented via the client automated assistant 236 of the computingdevice 218 and interface with the server device 202 that implementsother aspects of the automated assistant 204. The server device 202 canoptionally serve a plurality of users and their associated assistantapplications via multiple threads. In implementations where all or lessthan all aspects of the automated assistant 204 are implemented via aclient automated assistant 236 at the computing device 218, the clientautomated assistant 236 can be an application that is separate from anoperating system of the computing device 218 (e.g., installed “on top”of the operating system)—or can alternatively be implemented directly bythe operating system of the computing device 218 (e.g., considered anapplication of, but integral with, the operating system).

In some implementations, the automated assistant 204 and/or the clientautomated assistant 236 can include an input processing engine 206,which can employ multiple different modules for processing inputs and/oroutputs for the computing device 218 and/or the server device 202. Forinstance, the input processing engine 206 can include a speechprocessing module 208 that can process audio data received at anassistant interface 220 to identify the text embodied in the audio data.The audio data can be transmitted from, for example, the computingdevice 218 to the server device 202 in order to preserve computationalresources at the computing device 218.

The process for converting the audio data to text can include a speechrecognition algorithm, which can employ neural networks, and/orstatistical models for identifying groups of audio data corresponding towords or phrases. The text converted from the audio data can parsed by adata parsing module 210 and made available to the automated assistant astextual data that can be used to generate and/or identify commandphrases from the user. In some implementations, output data provided bythe data parsing module 210 can be provided to a parameter module 212 todetermine whether the user provided an input that corresponds to aparticular action and/or routine capable of being performed by theautomated assistant 204 and/or an application or agent that is capableof being accessed by the automated assistant 204. For example, assistantdata 216 can be stored at the server device 202 and/or the computingdevice 218, as client data 238, and can include data that defines one ormore actions capable of being performed by the automated assistant 204and/or client automated assistant 236, as well as parameters necessaryto perform the actions.

In some implementations, the system 200 can allow one or more users toemploy restrictions on particular interactions between passengers of avehicle and an automated assistant, and/or authorize such interactionsusing one or more different modalities. The system 200 can include avariety of different engines which can be embodied as executable data atone or more computing devices. The computing device 218 can beincorporated into a vehicle such as, but not limited to, a car, a truck,on airplane, an aerospace vehicle, a commercial vehicle, and/or anyother apparatus and that can be associated with a computing device. Avehicle that includes the computing device 218 can provide space for oneor more persons to ride in the vehicle. Furthermore, while one or morepersons are riding in the vehicle, the one or more users can interactwith the client automated assistant 236 and/or the automated assistant204 via one or more assistant interfaces 220. However, in someimplementations, the computing device 218 and/or the server device 202can be restricted according to a variety of contexts in which one ormore persons are interacting with the computing device 218 and/or theserver device 202.

In some implementations, the computing device 218, the automatedassistant, and/or the server device 202 can operate according todifferent operating modes. An operating mode can be selected by a modeengine 228, which can determine a suitable operating mode depending on acontext in which the computing device 218, the automated assistant,and/or the server device 202 are operating. For example, a contextualengine 230 can determine that a user is accessing a ride-sharingapplication via the computing device 218 while driving the vehicle. Thecontextual engine 230 can generate contextual data, and share thecontextual data with the mode engine 228. The mode engine 228 candetermine, based on the contextual data, to operate the computing device218 and/or the automated assistant according to a first operating mode.The first operating mode can be, for example, a restricted operatingmode in which any passenger in the vehicle will be unable to accessservices, actions, and/or information that is based on restricted datastored in the client data 238 and/or the assistant data 216.Furthermore, the first operating mode can be, for example a restrictedoperating mode in which the driver will be able to access services,actions, and/or information that is based on restricted data, and isotherwise accessible to the driver when not driving the vehicle infurtherance of ride-sharing activity. In this way, should a passenger orother guests to the vehicle attempt to invoke be client automatedassistant 236, the client automated assistant 236 can be responsivewithout providing an error message and/or otherwise involving undointeractions. For example, by allowing the first operating mode to beselected by the mode engine 228 without direct input, such as a spokenutterance, computational resources can be preserved as less inputs wouldbe processed (e.g., the user would not have to provide a request torestrict access to a passenger, every time a new passenger attempted toinvoke the automated assistant).

In some implementations, an assistant invocation task can effectuatedifferent operations of the automated assistant according to anoccupancy of the vehicle. For example, when the vehicle is exclusivelyoccupied by a person that is sitting in a seat for the driver, anybutton and/or other interface within the vehicle can provideunrestricted access to the automated assistant. However, when more thanone person is in the vehicle, an interface (e.g., a button) on thesteering wheel of the vehicle can provide unrestricted access to theautomated assistant, whereas other interfaces located closer topassengers' seats can provide restricted access to the automatedassistant. In this way, should a passenger invoke the automatedassistant by performing an assistant invocation task, such as pressing abutton on a passenger side door, the passenger would not be able tocause the automated assistant to access restricted data. Rather, thepassenger would have to wait for an unrestricted user to provideauthorization and/or the driver to interact with an interface on thesteering wheel, or other interface that is accessible to a driver of thevehicle while the driver is driving the vehicle.

In some implementations, one or more users of an automated assistant canrequest restrictions associated with their own particular account and/ortheir own particular data. As an example, when a user is driving thevehicle that includes the computing device 218, the user can provide aspoken utterance, and/or any other input, requesting that a restrictionbe generated. The spoken utterance can be processed at the inputprocessing engine 206, and the output generating engine 214 cancommunicate with a restriction engine 232. The restriction engine 232can determine, based on processing at the spoken utterance, one or morerestrictions to establish. For example, in some implementations, theuser can request that the automated assistant limit provisioning ofrestricted data when more than one person is in the vehicle. Occupancyof the vehicle can be determined by an occupancy engine 234, which canreceive sensor data from one or more sensors 224 of the vehicle. Thesensors 224 can include one or more of a touch interface, a microphone,a video camera, a tactile sensor, a weight sensor, an infrared sensor, amotion sensor, and/or any other sensor that can be used to detectoccupancy of an area or volume.

Based on the occupancy, determined by the occupancy engine 234, therestriction engine 232 can select one or more restrictions with which toenforce in order to limit interactions between one or more occupants ofthe vehicle and the computing device 218, an automated assistant, and/orthe server device 202. For example, in furtherance of the aforementionedexample, when the user is riding in the vehicle with another passenger,the occupancy engine 234 can determine that multiple persons are in thevehicle and the restriction engine 232 can, based on the occupancy,enforce the restriction of provisioning restricted data when multiplepersons are in the vehicle. In this way, each time the user gets in thevehicle with another passenger, they do not need to repeat restrictionrequests. Rather, the user can rely on occupancy being detected, and oneor more restrictions being enforced according to the occupancy. Forexample, if the user had set a private reminder regarding an event, andthe user is in the vehicle with another person when the reminder is setto go off, provisioning at the reminder can be limited at least withrespect to the restricted data that might be revealed. Therefore,instead of the computing device 218 rendering audio or visual datacharacterizing the reminder, the computing device 218 can render audiodata characterizing a notification sound, or render display datacharacterizing a graphical notification. Should the user want to bypassthe restriction they had previously set forth, the user can provideauthorization via the computing device 218, and/or one or more othercomputing devices 242.

For example, when the user is not driving the vehicle but is a passengerin the vehicle, the user can access another computing device 242, andprovide an input to an input device 244 of the other computing device242 in order to indicate authorization for provisioning the restricteddata. In some implementations, the other computing device 242 cantransmit authorization data to the computing device 218, which can beprocessed by an authorization engine 226, in order to confirm thatauthorization was provided by the user. In response to the authorizationengine 226 confirming that authorization was received from the user, theauthorization engine 226 can generate confirmation data, which can betransmitted to the other computing device 242. In response to the othercomputing device 242 receiving the confirmation data, the othercomputing device 242 can provide an indication via an output device 246of the other computing device 242. The indication can confirm to theuser that their authorization was accepted, and that the restrictionresponding to the recent notification will be bypassed for thatnotification. Alternatively, and/or additionally, the contextual engine230 can determine a context in which the user provided authorization,and permit the restriction to be bypassed so long as the context doesnot change. For example, if the user is riding to a particulardestination and/or is riding with one or more particular passengers, thedestination and/or the passengers can be characterized as a particularcontext. Therefore, when the user is leaving the particular destinationafter arriving, and/or the passengers leave the vehicle, or a differentpassenger enters the vehicle, the restriction can be enforced again, atleast until a separate authorization is provided by the user.

FIG. 3A and FIG. 3B illustrate method 300 and method 320 forestablishing and enforcing access restrictions for an automatedassistant based on one or more different factors. The method 300 and themethod 320 can be performed by one or more computing devices,applications, and/or any other apparatus or module capable of providingaccess to an automated assistant. The method 300 can include anoperation 302 of determining whether a spoken utterance was detected atan automated assistant interface of a computing device. In someimplementations, the automated assistant interface can be connected to avehicle computing device that is located within and/or otherwiseattached to a vehicle. One or more passengers of the vehicle caninteract with the automated assistant via the vehicle computing deviceusing one or more different inputs provided to one or more differentinput modalities available within the vehicle. The vehicle can be onethat is owned by a user who also owns multiple different computingdevices through which an automated assistant is accessible.Alternatively, the vehicle can be owned by a third party entity thatallows users to operate the vehicle through a paid service.Alternatively, or additionally, the vehicle can be employed by user forconducting ride-sharing activities, which can be tracked by anapplication that is accessible via the vehicle computing device and/or aseparate computing device owned by the user.

The method 300 can further include an optional operation 304 ofdetermining occupancy of the vehicle. Occupancy can refer to a number ofpersons that are located within the vehicle, and/or riding in thevehicle. Occupancy can be determined based on one or more sensor outputsof one or more sensors that are responsive to environmental changeswithin the vehicle. Therefore, as one or more persons enter and/or exitthe vehicle, occupancy can change and be characterized by the one ormore sensor outputs. The sensors can be in communication with thevehicle computing device and/or a remote server device, that is also incommunication with the vehicle computing device. Furthermore, permissionto determine occupancy can be granted or revoked by a user that owns thevehicle and/or any other passenger that occupies the vehicle.

The method 300 can further include an operation 306 of determiningwhether the spoken utterance was provided by a restricted user. Arestricted user can be a person that interacts with the automatedassistant but is not permitted to access restricted data via theautomated assistant. For example, when a passenger of a vehicle that isowned by a user provides the spoken utterance, the passenger can beconsidered a restricted user because the passenger does not own thevehicle. Alternatively, or additionally, the passenger can be arestricted user because the passenger is not listed on an account thatis accessible via, or otherwise associated with, the vehicle computingdevice.

When the spoken utterance is not provided by a restricted user, orotherwise provided by an unrestricted at user, the method 300 canproceed from the operation 306 to the operation 314. The operation 314can include identifying one or more requests embodied by the spokenutterance. For example, audio data corresponding to the spoken utterancecan be processed at the vehicle computing device and/or a server devicein order to determine natural language understanding of the spokenutterance. Data characterizing the natural language understanding can befurther processed to identify one or more requests that are embodied inthe spoken utterance. For example, content of the spoken utterance cancorrespond to a request for the automated assistant to perform one ormore actions, one or more routines, and/or any other operation capableof being performed by the automated assistant.

When the spoken utterance is provided by a restricted user, the method300 can proceed from the operation 306 to the operation 308. Theoperation 308 can include determining whether the vehicle is occupied bymultiple persons. In some implementations, determining whether thevehicle is occupied by multiple persons can be based on the occupancydetermined at the operation 304. Alternatively, or additionally,determining whether the vehicle is occupied by multiple persons can bedetermined using other information, such as voice characteristics of oneor more persons within the vehicle, client data provided via one or moreclient devices being carried by one or more persons within the vehicle,schedule data corresponding to one or more persons and indicatingwhether the one or more persons plan on riding in the vehicle, and/orany other information that can be used to determine occupancy of avehicle.

When the vehicle is not occupied by multiple persons, the method 300 canproceed from the operation 308 to the operation 314. When the vehicle isoccupied by multiple persons, the method 300 can proceed from theoperation 308 to the operation 310. The operation 310 can includeoperating in furtherance of receiving authorization from an unrestricteduser. The unrestricted user can be one that is associated with data thatwill be accessed in order to fulfill one or more requests that have beenprovided via the detected spoken utterance. Alternatively, oradditionally, the unrestricted user can be one that is permitted toaccess restricted data that the user who provided the spoken utteranceis attempting to access. In some implementations the operation 310 caninclude causing the vehicle computing device to await authorization fromthe unrestricted user. Alternatively, or additionally, the operation 310can include causing the vehicle computing device to render a prompt thatrequests the authorization from the unrestricted user.

The method 300 can proceed from the operation 310 to the operation 312,which can include determining whether authorization was received fromthe unrestricted user. When authorization is received from theunrestricted user, the method 300 can proceed from the operation 312 tothe operation 314. When the operation 314 is completed, the method 300can proceed from the operation 314 to the operation 316. The operation316 can include performing one or more actions in furtherance offulfilling the one or more requests. For instance, a restricted user canprovide the spoken utterance, and the spoken utterance can embody arequest for the automated assistant to perform a routine includingmultiple different actions. Therefore, when authorization is received atthe operation 312, and the request to perform the routine is identifiedat the operation 314, the automated assistant can operate in furtheranceof executing the one or more actions for filling the request to performthe routine.

When authorization is not received, and/or when one or more actions havebeen completed, the method 300 can proceed from continuation element “A”to the method 320, at operation 322. In other words, continuationelement A links the method 300 to the method 320, as illustrated in FIG.3A and FIG. 3B. In FIG. 3B, the method 320 can proceed from thecontinuation element “A” to the operation 322. The operation 322 caninclude determining whether an unrestricted user provided otherinstructions. The other instructions can include, but are not limited toone or more requests for setting restrictions on access to services,data, and/or any other feature that can be accessed via the vehiclecomputing device, and/or any other computing device associated with theunrestricted user. For example, if the unrestricted user did not provideauthorization, the unrestricted user can nonetheless provide one or morerequests for other restrictions to be enforced. Such restrictions can bebased on context, types of request, particular users making the request,and/or any other property that can be associated with interactions withan automated assistant.

When the unrestricted user is determined to have provided otherinstructions, the method 320 can proceed from the operation 322 to theoperation 326. The operation 326 can include modifying one or moreaccess settings in accordance with the other instructions. When theunrestricted user is determined to have not provided other instructions,the method 320 can proceed from the operation 322 to the operation 324.The operation 324 can include bypassing identifying one or more requestsembodied in the spoken utterance. In other words, because the user whoprovided the one of my request was not authorized to make such requests,the method 320 can include bypassing further processing of the spokenutterance, and/or subsequent inputs, from the user. The method 320 canproceed from the operation 324 and/or the operation 326, viacontinuation element “B,” to operation 302. As illustrated in FIG. 3Band FIG. 3A, the continuation element “B” can provide a link between themethod 320 and the method 300. Furthermore, and via continuation element“B,” the method 320 can return to determining whether a subsequentspoken utterance has been received at the operation 302.

FIG. 4 is a block diagram of an example computer system 410. Computersystem 410 typically includes at least one processor 414 whichcommunicates with a number of peripheral devices via bus subsystem 412.These peripheral devices may include a storage subsystem 424, including,for example, a memory 425 and a file storage subsystem 426, userinterface output devices 420, user interface input devices 422, and anetwork interface subsystem 416. The input and output devices allow userinteraction with computer system 410. Network interface subsystem 416provides an interface to outside networks and is coupled tocorresponding interface devices in other computer systems.

User interface input devices 422 may include a keyboard, pointingdevices such as a mouse, trackball, touchpad, or graphics tablet, ascanner, a touch screen incorporated into the display, audio inputdevices such as voice recognition systems, microphones, and/or othertypes of input devices. In general, use of the term “input device” isintended to include all possible types of devices and ways to inputinformation into computer system 410 or onto a communication network.

User interface output devices 420 may include a display subsystem, aprinter, a fax machine, or non-visual displays such as audio outputdevices. The display subsystem may include a cathode ray tube (CRT), aflat-panel device such as a liquid crystal display (LCD), a projectiondevice, or some other mechanism for creating a visible image. Thedisplay subsystem may also provide non-visual display such as via audiooutput devices. In general, use of the term “output device” is intendedto include all possible types of devices and ways to output informationfrom computer system 410 to the user or to another machine or computersystem.

Storage subsystem 424 stores programming and data constructs thatprovide the functionality of some or all of the modules describedherein. For example, the storage subsystem 424 may include the logic toperform selected aspects of method 300 and method 320, and/or toimplement one or more of computing device 114, vehicle 112, serverdevice 126, system 200, server device 202, computing device 218, othercomputing device 242, and/or any other apparatus, module, and/or enginediscussed herein.

These software modules are generally executed by processor 414 alone orin combination with other processors. Memory 425 used in the storagesubsystem 424 can include a number of memories including a main randomaccess memory (RAM) 430 for storage of instructions and data duringprogram execution and a read only memory (ROM) 432 in which fixedinstructions are stored. A file storage subsystem 426 can providepersistent storage for program and data files, and may include a harddisk drive, a floppy disk drive along with associated removable media, aCD-ROM drive, an optical drive, or removable media cartridges. Themodules implementing the functionality of certain implementations may bestored by file storage subsystem 426 in the storage subsystem 424, or inother machines accessible by the processor(s) 414.

Bus subsystem 412 provides a mechanism for letting the variouscomponents and subsystems of computer system 410 communicate with eachother as intended. Although bus subsystem 412 is shown schematically asa single bus, alternative implementations of the bus subsystem may usemultiple busses.

Computer system 410 can be of varying types including a workstation,server, computing cluster, blade server, server farm, or any other dataprocessing system or computing device. Due to the ever-changing natureof computers and networks, the description of computer system 410depicted in FIG. 4 is intended only as a specific example for purposesof illustrating some implementations. Many other configurations ofcomputer system 410 are possible having more or fewer components thanthe computer system depicted in FIG. 4 .

In situations in which the systems described herein collect personalinformation about users (or as often referred to herein,“participants”), or may make use of personal information, the users maybe provided with an opportunity to control whether programs or featurescollect user information (e.g., information about a user's socialnetwork, social actions or activities, profession, a user's preferences,or a user's current geographic location), or to control whether and/orhow to receive content from the content server that may be more relevantto the user. Also, certain data may be treated in one or more waysbefore it is stored or used, so that personal identifiable informationis removed. For example, a user's identity may be treated so that nopersonal identifiable information can be determined for the user, or auser's geographic location may be generalized where geographic locationinformation is obtained (such as to a city, ZIP code, or state level),so that a particular geographic location of a user cannot be determined.Thus, the user may have control over how information is collected aboutthe user and/or used.

In some implementations, a method implemented by one or more processorsis set forth as including operations such as receiving, from a vehiclecomputing device, input data in response to a spoken utterance receivedat an automated assistant interface of a vehicle computing device,wherein the vehicle computing device is located in a vehicle with afirst user and a second user. The method can further includedetermining, in furtherance of responding to the spoken utterance,whether an assistant invocation task was performed by the second user toovercome a restriction on access, by the first user, to restricted datathat is otherwise accessible to the second user via the automatedassistant, wherein the vehicle computing device causes the automatedassistant to be invoked in response to the second user performing one ormore assistant invocation tasks that include the assistant invocationtask. The method can further include, when the assistant invocation taskwas not performed by the second user: causing, in response todetermining that the assistant invocation task was not performed by thesecond user, the vehicle computing device to operate in furtherance ofreceiving, from the second user, authorization to render output that isbased on the restricted data, wherein the second user providesauthorization to render the output by performing a particular assistantinvocation task of the one or more assistant invocation tasks. Themethod can further include determining, subsequent to causing thevehicle computing device to operate in furtherance of receivingauthorization to render the output, whether authorization was receivedfrom the second user. The method can further include, when authorizationis determined to have been received from the second user: causing thevehicle computing device to render the output that is based on therestricted data and the input data.

In some implementations, the method can further include, prior tocausing the vehicle computing device to render the output: determiningan occupancy of the vehicle based on vehicle data that is accessible tothe vehicle computing device. In some implementations, the method canfurther include causing, based on the occupancy, the automated assistantand/or the vehicle computing device to operate according to an operatingmode, selected from two or more operating modes. In someimplementations, determining whether the assistant invocation task wasperformed by the second user is executed responsive to the vehiclecomputing device being in the operating mode. In some implementations,the vehicle data is based on output from one or more sensors that areconnected to the vehicle and are responsive to changes in occupancy ofthe vehicle. In some implementations, the vehicle data is provided by anapplication that is accessible via the vehicle computing device, and thevehicle data indicates whether occupancy within the vehicle is infurtherance of a ride-sharing activity. In some implementations, themethod can further include, when the assistant invocation task isperformed by the second user: providing, based on the second userperforming the assistant invocation task, access to the restricted datavia the vehicle computing device, and causing the vehicle computingdevice to render the output that is based on the restricted data.

In some implementations, determining whether authorization was receivedfrom the second user includes: determining whether the second userprovided a separate spoken utterance to the automated assistantinterface and/or whether the second user provided an input to a separateinterface that is connected to the vehicle computing device. In someimplementations, the method can further include, when the authorizationis determined to have not been received from the second user: causingthe vehicle computing device to render a separate output that indicatesa restriction of the automated assistant for responding to the spokenutterance. In some implementations, the method can further include, whenthe authorization is determined to have not been received from thesecond user: generating responsive data that characterizes a request forthe second user to provide another spoken utterance that is differentfrom the spoken utterance previously provided by the first user, andcausing the vehicle computing device to render the request to the seconduser. In some implementations, the method can further include, when theauthorization is determined to have not been received from the seconduser: determining that the second user has provided a limitation requestto the automated assistant, wherein the limitation request specifies oneor more limitations on interactions between the first user and theautomated assistant, and generating, in response to determining that thesecond user has provided the limitation request to the automatedassistant, restriction data that characterizes the one or morelimitations on interactions between the first user and the automatedassistant.

In some implementations, the one or more limitations characterized bythe restriction data includes a limitation on accessibility of theautomated assistant to the first user during a current route throughwhich the vehicle is driving, or until the vehicle arrives at aparticular destination. In some implementations, the one or morelimitations characterized by the restriction data include a limitationon accessibility to one or more types of data to the first user during acurrent route through which the vehicle is traveling, or until thevehicle arrives at a particular destination. In some implementations,the one or more limitations characterized by the restriction dataincludes a limitation on one or more types of commands that theautomated assistant can be responsive to when the one or more types ofcommands are provided by the first user during a current route throughwhich the vehicle is traveling, or until the vehicle arrives at aparticular destination. In some implementations, the input datacharacterizes a request for the automated assistant to perform anautomated assistant routine, which corresponds to two or more actionthat are performed by the automated assistant in response to one or moreusers providing the request to the automated assistant interface.

In some implementation, first responsive data and second responsive dataare generated in response to the automated assistant initializingperformance of a first action of the two or more actions correspondingto the automated assistant routine. Furthermore, the method can furtherinclude, subsequent to causing the vehicle computing device to renderthe output: causing the vehicle computing device to render a separateoutput according to a second action of the two or more actions infurtherance of completing the automated assistant routine. In someimplementations, the method can further include: generating responsivedata based on the input data, and, when the assistant invocation task isdetermined to have been performed by the second user: causing thevehicle computing device to render the separate output using theresponsive data.

In other implementations, a method implemented by one or more processorsis set forth as including operations such as processing input datacorresponding to a spoken utterance that is received at an automatedassistant interface of a vehicle computing device, wherein the vehiclecomputing device is located within a vehicle and the spoken utterancewas received when the vehicle is traveling to reach a destination. Themethod can further include determining, while the vehicle is travelingto reach the destination, an occupancy of the vehicle, wherein theoccupancy is determined based on output of one or more sensors that arein communication with the vehicle computing device. The method canfurther include determining, based on processing the input data, whetherthe spoken utterance includes a request for the automated assistant toperform at least one action that involves accessing restricted data thatis accessible to an unrestricted user via the vehicle computing device.The method can further include determining, based on the occupancy ofthe vehicle, whether the vehicle is currently occupied by theunrestricted user. The method can further include, when the spokenutterance includes the request for the automated assistant to perform atleast one action that involves accessing the restricted data and thevehicle is occupied by the unrestricted user: operating in furtheranceof receiving authorization from the unrestricted user. In someimplementations, the method can further include, when the authorizationis received from the unrestricted user via an authorization input to thevehicle computing device and/or another computing device: causing theautomated assistant to perform at least the one action that involvesaccessing the restricted data. In some implementations, the method canfurther include when the spoken utterance includes the request for theautomated assistant to perform at least one action that involvesaccessing the restricted data and the vehicle is not occupied by theunrestricted user: identifying at least one other action that does notinvolve accessing the restricted data, and causing the automatedassistant to perform the at least one other action in furtherance offulfilling the request.

In some implementations, the method can further include, when the spokenutterance includes the request for the automated assistant to perform atleast one action that involves accessing the restricted data and thevehicle is occupied by the unrestricted user, and when the authorizationis not received from the user via the vehicle computing device and/oranother computing device: identifying at least the one other action thatdoes not involve accessing the restricted data, and causing theautomated assistant to perform the at least one other action infurtherance of fulfilling the request. In some implementations,identifying the at least one other action includes identifying aparticular action that involves accessing unrestricted data, and therequests is a routine request that corresponds to an automated assistantroutine that involves two or more actions being performed by theautomated assistant.

In yet other implementations, a method implemented by one or moreprocessors is set forth as including operations such as processing inputdata corresponding to a spoken utterance that is received at anautomated assistant interface of a vehicle computing device, wherein thevehicle computing device is located within a vehicle and the spokenutterance is received from a user when the vehicle was traveling toreach a destination. In some implementations, the method can furtherinclude determining, based on processing the input and while the vehicleis traveling to reach the destination, whether restriction dataaccessible to the vehicle computing device characterizes a contextualrestriction that corresponds to a current context in which the spokenutterance was received at the automated assistant interface. In someimplementations, the method can further include, when the restrictiondata does not characterize the contextual restriction that correspondsto the current context in which the spoken utterance was received:causing the automated assistant to operate in furtherance of fulfillingone or more requests specified by the user via the spoken utterance. Insome implementations, the method can further include, when therestriction data characterizes the contextual restriction thatcorresponds to the current context in which the spoken utterance wasreceived: determining whether an authorization input was received froman unrestricted user for authorizing the automated assistant to operatein furtherance of fulfilling one or more requests specified via thespoken utterance. In some implementations, the method can furtherinclude, when the authorization input is determined to have not beenreceived from the unrestricted user: causing the automated assistant todeny access to restricted data for the user.

In some implementations, wherein determining whether the authorizationinput was received from the unrestricted user includes: determiningwhether a touch input was received at an interface of the vehiclecomputing device by the unrestricted user, wherein the interface of thevehicle computing device is located within the vehicle.

In yet other implementations, a method implemented by one or moreprocessors is set forth as including operations such as processing inputdata corresponding to a spoken utterance that is received at anautomated assistant interface of a vehicle computing device, wherein thevehicle computing device is located within a vehicle and the spokenutterance is received from a user when the vehicle was traveling toreach a destination. The method can further include determining, basedon processing the input and while the vehicle is traveling to reach thedestination, whether restriction data accessible to the vehiclecomputing device characterizes a contextual restriction that correspondsto a current context in which the spoken utterance was received at theautomated assistant interface. The method can further include, when therestriction data characterizes the contextual restriction thatcorresponds to the current context in which the spoken utterance wasreceived: determining whether an authorization input was received froman unrestricted user for authorizing the automated assistant to operatein furtherance of fulfilling one or more requests specified via thespoken utterance. In some implementations, the method can furtherinclude, when the authorization input is determined to have beenreceived from the unrestricted user: causing the automated assistant tooperate in furtherance of fulfilling the one or more requests specifiedvia the spoken utterance.

In some implementations, the method can further include, when therestriction data characterizes the contextual restriction thatcorresponds to the current context in which the spoken utterance wasreceived, and when the authorization input is determined to have beenreceived from the unrestricted user: causing the automated assistant tooperate to allow access, by the user, to the restricted data while atleast the current context satisfies the contextual restriction. In someimplementations, the method can further include, when the restrictiondata characterizes the contextual restriction that corresponds to thecurrent context in which the spoken utterance was received, and when theauthorization input is determined to have been received from theunrestricted user: causing the automated assistant to be responsive toone or more subsequent spoken utterances from the user while at leastthe current context satisfies the contextual restriction, wherein theautomated assistant accesses the restricted data in response to the oneor more spoken utterances from the user.

While several implementations have been described and illustratedherein, a variety of other means and/or structures for performing thefunction and/or obtaining the results and/or one or more of theadvantages described herein may be utilized, and each of such variationsand/or modifications is deemed to be within the scope of theimplementations described herein. More generally, all parameters,dimensions, materials, and configurations described herein are meant tobe exemplary and that the actual parameters, dimensions, materials,and/or configurations will depend upon the specific application orapplications for which the teachings is/are used. Those skilled in theart will recognize, or be able to ascertain using no more than routineexperimentation, many equivalents to the specific implementationsdescribed herein. It is, therefore, to be understood that the foregoingimplementations are presented by way of example only and that, withinthe scope of the appended claims and equivalents thereto,implementations may be practiced otherwise than as specificallydescribed and claimed. Implementations of the present disclosure aredirected to each individual feature, system, article, material, kit,and/or method described herein. In addition, any combination of two ormore such features, systems, articles, materials, kits, and/or methods,if such features, systems, articles, materials, kits, and/or methods arenot mutually inconsistent, is included within the scope of the presentdisclosure.

We claim:
 1. A method implemented by one or more processors, the methodcomprising: processing input data corresponding to a spoken utterance,of a user, that is received at an automated assistant interface of avehicle computing device, wherein the vehicle computing device islocated within a vehicle and the spoken utterance is received from theuser while the vehicle is traveling to a destination location;determining, based on processing the input data and while the vehicle istraveling to the destination location, whether the user is a restricteduser or an unrestricted user that are both located in the vehicle; inresponse to determining that the user is a restricted user: causing theautomated assistant to refrain from fulfilling one or more requestsspecified via the spoken utterance; and causing the automated assistantto provide output that indicates the automated assistant has refrainedfrom fulfilling the one or more requests specified via the spokenutterance; processing additional input data corresponding to anadditional spoken utterance, of the unrestricted user, that is receivedat the automated assistant interface of the vehicle computing device;generating, based on processing the additional input data, restrictiondata that characterizes a contextual restriction that corresponds to acurrent context in which the spoken utterance, of the user, was receivedat the automated assistant interface; and in response to determiningthat the restriction data that characterizes the contextual restrictionthat corresponds to the current context in which the spoken utterancewas received: determining whether the contextual restriction authorizedthe automated assistant to operate in furtherance of fulfilling the oneor more requests specified via the spoken utterance, and in response todetermining that the contextual restriction authorized the automatedassistant to operate in furtherance of fulfilling the one or morerequests specified via the spoken utterance: causing the automatedassistant to operate in furtherance of fulfilling the one or morerequests specified via the spoken utterance.
 2. The method of claim 1,wherein the unrestricted user is driving the vehicle.
 3. The method ofclaim 2, wherein the restricted user is a passenger of the vehicle thatis not driving the vehicle.
 4. The method of claim 3, wherein therestricted user that provided the spoken utterance is participating in aride sharing activity.
 5. The method of claim 4, wherein the contextualrestriction is enforced throughout a duration of the ride sharingactivity and until the vehicle arrives at the destination location.
 6. Asystem comprising: at least one processor; and memory storinginstructions that, when executed, cause the at least one processor toperform operations, the operations comprising: processing input datacorresponding to a spoken utterance of a user that is received at anautomated assistant interface of a vehicle computing device, wherein thevehicle computing device is located within a vehicle and the spokenutterance is received from the user while the vehicle is traveling to adestination location; determining, based on processing the input dataand while the vehicle is traveling to the destination location, whetherthe user is a restricted user or an unrestricted user that are bothlocated in the vehicle; in response to determining that the user is arestricted user: causing the automated assistant to refrain fromfulfilling one or more requests specified via the spoken utterance; andcausing the automated assistant to provide output that indicates theautomated assistant has refrained from fulfilling the one or morerequests specified via the spoken utterance; processing additional inputdata corresponding to an additional spoken utterance, of theunrestricted user, that is received at the automated assistant interfaceof the vehicle computing device; generating, based on processing theadditional input data, restriction data that characterizes a contextualrestriction that corresponds to a current context in which the spokenutterance, of the user, was received at the automated assistantinterface; and in response to determining that the restriction data thatcharacterizes the contextual restriction that corresponds to the currentcontext in which the spoken utterance was received: determining whetherthe contextual restriction authorized the automated assistant to operatein furtherance of fulfilling the one or more requests specified via thespoken utterance, and in response to determining that contextualrestriction authorized the automated assistant to operate in furtheranceof fulfilling the one or more requests specified via the spokenutterance: causing the automated assistant to operate in furtherance offulfilling the one or more requests specified via the spoken utterance.7. The system of claim 6, wherein the unrestricted user is driving thevehicle.
 8. The system of claim 7, wherein the restricted user is apassenger of the vehicle that is not driving the vehicle.
 9. The systemof claim 8, wherein the restricted user that provided the spokenutterance is participating in a ride sharing activity.
 10. The system ofclaim 9, wherein the contextual restriction is enforced throughout aduration of the ride sharing activity and until the vehicle arrives atthe destination location.
 11. A non-transitory computer-readable storagemedium storing instructions that, when executed, cause at least oneprocessor to perform operations, the operations comprising: processinginput data corresponding to a spoken utterance of a user that isreceived at an automated assistant interface of a vehicle computingdevice, wherein the vehicle computing device is located within a vehicleand the spoken utterance is received from the user while the vehicle istraveling to a destination location; determining, based on processingthe input data and while the vehicle is traveling to the destinationlocation, whether the user is a restricted user or an unrestricted userthat are both located in the vehicle; in response to determining thatthe user is a restricted user: causing the automated assistant torefrain from fulfilling one or more requests specified via the spokenutterance; and causing the automated assistant to provide output thatindicates the automated assistant has refrained from fulfilling the oneor more requests specified via the spoken utterance; processingadditional input data corresponding to an additional spoken utterance,of the unrestricted user, that is received at the automated assistantinterface of the vehicle computing device; generating, based onprocessing the additional input data, restriction data thatcharacterizes a contextual restriction that corresponds to a currentcontext in which the spoken utterance, of the user, was received at theautomated assistant interface; and in response to determining that therestriction data that characterizes the contextual restriction thatcorresponds to the current context in which the spoken utterance wasreceived: determining whether the contextual restriction authorized theautomated assistant to operate in furtherance of fulfilling the one ormore requests specified via the spoken utterance, and in response todetermining that the contextual restriction authorized the automatedassistant to operate in furtherance of fulfilling the one or morerequests specified via the spoken utterance: causing the automatedassistant to operate in furtherance of fulfilling the one or morerequests specified via the spoken utterance.
 12. The non-transitorycomputer-readable storage medium of claim 11, wherein the unrestricteduser is driving the vehicle.
 13. The non-transitory computer-readablestorage medium of claim 12, wherein the restricted user is a passengerof the vehicle that is not driving the vehicle.